PER-8985 feat: Playwright drop-in baseline seeding — build attributes + exec orchestration by Shivanshu-07 · Pull Request #2335 · percy/cli

Shivanshu-07 · 2026-07-13T13:03:09Z

What

CLI-side support for the @percy/playwright toHaveScreenshot drop-in's baseline flow:

@percy/client — createBuild accepts allow-listed drop-in build sources via PERCY_BUILD_SOURCE (playwright-dropin / playwright-dropin-baseline) plus dropin-baseline-candidate / dropin-baseline-setup attributes (env or explicit options). The API keys baseline auto-approval on these.
@percy/cli-exec — generic baseline-provider discovery: any installed SDK package can declare "@percy/cli": { "baselineProvider": "..." } (no framework code enters the CLI, same scaling model as command discovery). On an EMPTY project, percy exec uploads the committed baseline screenshots as an auto-approved build ⬆️ Bump @babel/register from 7.10.4 to 7.10.5 #1 before the user's command runs (the suite never executes for the baseline), then starts the head build. Established projects get a notice pointing at the SDK's explicit playwright:setup-baseline command — the API's no-build-persisted sentinel guarantees no stray builds. Seeding dispatches by project type: web → rendered web snapshots (root DOM + image resource, the percy upload shape); app → comparison ingest (tag + tile, no render).
@percy/core — the server-decided build source is kept on percy.build so SDKs can key on it through the healthcheck.

Depends on

PER-8985 feat: toHaveScreenshot drop-in — route existing Playwright assertions through Percy percy-playwright#640 (the SDK provider + command)
percy/percy-api#6178 + companion API PR (server-side decision + auto-approval)

Testing

Client + exec suites green incl. 11 new seeding specs; E2E verified against a local percy-api and on staging (empty web/app seed → head diff flow, established-project sentinel, explicit setup command).

Deploy ordering

percy-api (percy/percy-api#6527) must ship first. Against an API that predates the dropin-baseline-candidate attribute, the compat guard abandons the seed build unused — on an established project that leaves one stray unfinalized build per run until the API side is live (they expire server-side, but numbering inflates).

🤖 Generated with Claude Code

…orchestration - @percy/client createBuild: allow-listed drop-in build sources via PERCY_BUILD_SOURCE, plus dropin-baseline-candidate / dropin-baseline-setup attributes (env or explicit options) the API keys baseline auto-approval on. - @percy/cli-exec: generic baseline-provider discovery (any installed SDK package can declare "@percy/cli".baselineProvider). On an empty project, percy exec uploads the committed baseline screenshots as an auto-approved build #1 before the user's command runs; established projects get a notice pointing at the SDK's explicit setup command. Seeding helpers exported for SDK-contributed commands. - @percy/core: expose the server-decided build source on percy.build so SDKs can key on it through the healthcheck. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Verified E2E against a local percy-api: a web project's snapshot create requires a root resource, so raw-PNG comparison tiles are rejected. Seed uploads now use client.sendSnapshot with the same generated root-DOM + image-resource shape 'percy upload' uses for web projects — Percy renders the baseline in the project's own browsers, so it pairs with the head run's DOM snapshots by (name, browser, width). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Seeding dispatches by project type: web projects seed rendered snapshots (root DOM + image resource); app projects seed raw comparison uploads (tag + tile), exactly how App Percy ingests screenshots. Exec seeding now engages for app tokens too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The same env var the SDK override honors — one switch turns off both the toHaveScreenshot override and exec's baseline provider discovery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- core: only set percy.build.source when the API returned one — an own 'source: undefined' key survives in-process but drops out of JSON responses, breaking the API-server shape specs. - cli-exec: cover the exec seeding block (real provider-package fixture) and the remaining baseline.js branches for the 100% coverage gate. - cli-exec: sanitize dirent names / paths at the fs boundary in the provider walk (also clears semgrep's path-join-resolve-traversal findings on the diff). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Direct sanitizer unit tests, walk fixtures for every discovery skip branch (empty dir, ESM named-export provider without discoverBaselines), and the non-web/app token exec branch; istanbul-ignore the two defense-in-depth dirent guards unreachable via readdir. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The seed build keeps rendering and only auto-approves once it reaches finished; head snapshots select their baseline as they process, so starting the head immediately raced the approval and a lost race made the whole first run report as new instead of diffing. Poll the seed build up to the pipeline latency budget (10m) with a 30s heartbeat, degrade to a warning instead of blocking on timeout or status errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…pin-baseline # Conflicts: # packages/client/src/client.js # packages/client/test/client.test.js

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Master-inherited coverage gap from the metadata-SSRF change — the catch path for a colon-containing host that fails URL parsing was never exercised, holding @percy/core under the 100% gate. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Every caller (matchMetadataHost) null-guards before calling, so the branch cannot be exercised by tests; it held @percy/core under the 100% gate on master too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The seed upload omitted browserName, so seed and head comparisons landed on different canonical tag rows and screenshot pairing missed — the first app run diffed nothing. One tag shape, defined by the SDK. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

- discovery walk now mirrors @percy/cli's findModulePackages exactly: stop at the nearest node_modules, never cross the home directory, and degrade to [] on any filesystem error — a throw here previously escaped to exec's outer catch and disabled Percy for the whole run - provider modules are confined to their declaring package (a baselineProvider path escaping the package root is skipped) - only ever seed build #1: an API that predates the candidate attribute hands back a normal build, which previously seeded stray baseline builds into established projects on every run - falsy provider baseline entries are filtered instead of silently terminating an upload worker Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… idiom) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Shivanshu-07 · 2026-07-17T08:39:44Z

Claude Code PR Review

PR: #2335 • Head: 0a38a90 • Reviewers: fallback inline checklist (deep-review agent)

Summary

CLI-side drop-in baseline seeding: generic baselineProvider package discovery, seed-build creation/upload (percy exec seeds an empty project's build #1 from committed screenshots, then waits for it to finish before the head build starts), new allow-listed build source + candidate/setup attributes in @percy/client, plus two @percy/core coverage fixes inherited from master.

Review Table

Priority	Category	Check	Status	Notes
High	Security	No hardcoded secrets or credentials	Pass
High	Security	Authentication/authorization checks present	N/A	token handled by @percy/client env
High	Security	Input validation and sanitization	Pass	sources allow-listed (DROPIN_BUILD_SOURCES); env flags strict `=== 'true'`
High	Security	No IDOR — resource ownership validated	N/A
High	Security	No SQL injection (parameterized queries)	N/A
High	Correctness	Logic is correct, handles edge cases	Pass	2 High findings fixed in `0a38a90` lineage: discovery walk now degrades to [] on fs errors (a throw previously escaped to exec's catch and disabled Percy for the run); seeding is bounded to build #1 so a pre-candidate API can no longer accrete stray baseline builds on established projects
High	Correctness	Error handling is explicit, no swallowed exceptions	Pass	never-throw seeding posture is by design; walk failures logged at debug
High	Correctness	No race conditions or concurrency issues	Pass	seed-wait closes the head-baseline race; parallel shards skip seeding
Medium	Testing	New code has corresponding tests	Pass	walk-error, containment-escape, homedir-bound, non-first-build, falsy-entry cases added
Medium	Testing	Error paths and edge cases tested	Pass
Medium	Testing	Existing tests still pass (no regressions)	Pass	all package suites + 100% coverage gates green
Medium	Performance	No N+1 queries or unbounded data fetching	Pass	walk now stops at the nearest node_modules (was: every ancestor to /)
Medium	Performance	Long-running tasks use background jobs	N/A
Medium	Quality	Follows existing codebase patterns	Pass	walk mirrors @percy/cli findModulePackages semantics exactly
Medium	Quality	Changes are focused (single concern)	Pass	the two @percy/core fixes unblock master's own red Test workflow
Low	Quality	Meaningful names, no dead code	Pass
Low	Quality	Comments explain why, not what	Pass
Low	Quality	No unnecessary dependencies added	Pass

Findings

All review findings were fixed on this PR before this report:

High packages/cli-exec/src/baseline.js: the node_modules walk could throw (EACCES / node_modules-as-file) out of findBaselineProvider into exec's outer catch — silently disabling Percy for the whole run. Walk now try/caught → [], matching the command-discovery pattern.
High packages/cli-exec/src/baseline.js: against an API that predates the candidate attribute, every run would seed a stray "baseline" build into established projects. Seeding now requires build-number === 1.
Medium waitForSeedBuild uses client.getBuild — verified working with standard write project tokens against percy-api (staging E2E), so the read-token doc caveat does not bite; kept as-is.
Low: walk bounded at the home directory and stops at the nearest node_modules (narrower import surface); provider modules confined to their declaring package; falsy provider entries filtered instead of terminating an upload worker; containment paths wrapped in the sanctioned sanitizer idiom (semgrep).

Verdict: PASS

…pin-baseline # Conflicts: # packages/core/src/utils.js

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…tice An all-uploads-failed seed previously finalized an EMPTY auto-approved build #1, establishing the project with a blank baseline and permanently disabling the auto-seed path. Zero uploads now skip finalize (the unfinalized build expires server-side) and point at setup-baseline. Also: the established-project notice is one line instead of two, and a failed status poll reports 'state: unknown' instead of 'undefined'. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…ower walk - exec drives maybeSeedBaseline through yieldTo so the up-to-10min seed wait stays tick-responsive to Ctrl-C (it was the one multi-minute stretch the abort signal couldn't unwind) - the zero-seed warning no longer promises that a plain re-run recovers (an unfinalized candidate build can bump the build number past the compat guard) — setup-baseline is the recovery path - unscoped walk collection narrowed from percy-* to percy-cli-*, matching findModulePackages exactly - env-fallback test added for the setup attribute Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Shivanshu-07 · 2026-07-27T11:24:44Z

Claude Code PR Review

PR: #2335 • Head: da65d31 • Reviewers: fallback inline checklist (deep-review agent, re-review)

Summary

Re-review after the master merge, reviewer-feedback fixes (zero-upload seed no longer finalizes), and comment cleanup. CLI-side drop-in baseline seeding: provider discovery, seed build #1 from committed screenshots with a bounded wait before the head build, allow-listed build sources, and the playwright:setup-baseline command.

Review Table

Priority	Category	Check	Status	Notes
High	Security	No hardcoded secrets or credentials	Pass
High	Security	Authentication/authorization checks present	N/A	token handled by @percy/client env
High	Security	Input validation and sanitization	Pass	env flags strict `=== 'true'`; sources allow-listed; rogue provider buildSource cannot inject
High	Security	No IDOR — resource ownership validated	N/A
High	Security	No SQL injection (parameterized queries)	N/A
High	Correctness	Logic is correct, handles edge cases	Pass	zero-seed path verified (no finalize, no wait, never throws); walk narrowed to percy-cli-* matching findModulePackages
High	Correctness	Error handling is explicit, no swallowed exceptions	Pass	never-throw seeding posture holds on every path
High	Correctness	No race conditions or concurrency issues	Pass	seed-wait now tick-responsive to Ctrl-C via yieldTo (fixed in `da65d31`); worker queue race-free
Medium	Testing	New code has corresponding tests	Pass	zero-seed, established sentinel, pre-candidate guard, walk bounds, env fallbacks (setup variant added in `da65d31`)
Medium	Testing	Error paths and edge cases tested	Pass
Medium	Testing	Existing tests still pass (no regressions)	Pass	all package suites + 100% coverage gates green (Windows core flake was a degraded runner on a master-owned spec)
Medium	Performance	No N+1 queries or unbounded data fetching	Pass	walk stops at nearest node_modules, home-dir bound
Medium	Performance	Long-running tasks use background jobs	N/A
Medium	Quality	Follows existing codebase patterns	Pass	HTML shape mirrors percy upload; walk mirrors command discovery
Medium	Quality	Changes are focused (single concern)	Pass	deploy-ordering note (api PR first) added to the PR description
Low	Quality	Meaningful names, no dead code	Pass
Low	Quality	Comments explain why, not what	Pass	zero-seed recovery text corrected (setup-baseline, not re-run)
Low	Quality	No unnecessary dependencies added	Pass

Findings

Re-review findings (2 Medium, 3 Low), all fixed on this PR in da65d31:

Medium packages/cli-exec/src/exec.js: the up-to-10-minute seed wait was the one stretch the abort signal couldn't unwind — a first Ctrl-C appeared ignored. Now driven through yieldTo, keeping the runner tick-responsive.
Medium packages/cli-exec/src/baseline.js: the zero-seed warning promised a plain re-run recovers, but a stray unfinalized candidate build can bump the build number past the pre-candidate compat guard — recovery text now points at percy playwright:setup-baseline, and the guard's comment states its actual role.
Low: unscoped walk collection narrowed from percy-* to percy-cli-* (matches findModulePackages, smaller auto-import surface); deploy-ordering requirement documented in the PR body; env-fallback test added for the setup attribute.

Earlier human-review threads (aryanku-dev): all resolved — write-token wait verified empirically on staging; zero-upload finalize fixed; established-project notice reduced to one line.

Verdict: PASS

Shivanshu-07 and others added 3 commits July 13, 2026 18:31

Shivanshu-07 requested a review from a team as a code owner July 13, 2026 13:03

Shivanshu-07 requested review from aryanku-dev and prklm10 July 13, 2026 13:03

github-advanced-security AI found potential problems Jul 13, 2026

View reviewed changes

Shivanshu-07 and others added 12 commits July 14, 2026 10:28

feat: PERCY_DROPIN_DISABLE opts out of baseline seeding

8ba3083

The same env var the SDK override honors — one switch turns off both the toHaveScreenshot override and exec's baseline provider discovery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/master' into feat/playwright-dro…

ee87aa4

…pin-baseline # Conflicts: # packages/client/src/client.js # packages/client/test/client.test.js

test: cover the pending-state branch of waitForSeedBuild

174dd3a

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

test: cover findBaselineProvider default-parameter branches

b5609a5

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

test: cover the healthcheck build source branch

ed1d400

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chore: istanbul-ignore canonicalHost's unreachable null guard

25e746a

Every caller (matchMetadataHost) null-guards before calling, so the branch cannot be exercised by tests; it held @percy/core under the 100% gate on master too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

github-advanced-security AI found potential problems Jul 17, 2026

View reviewed changes

Comment thread packages/cli-exec/src/baseline.js Fixed

Comment thread packages/cli-exec/src/baseline.js Fixed

chore: wrap provider containment paths in the path sanitizer (semgrep…

0a38a90

… idiom) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/master' into feat/playwright-dro…

b73530f

…pin-baseline # Conflicts: # packages/core/src/utils.js

aryanku-dev reviewed Jul 23, 2026

View reviewed changes

Comment thread packages/cli-exec/src/baseline.js

Comment thread packages/cli-exec/src/baseline.js

Comment thread packages/cli-exec/src/baseline.js

Shivanshu-07 and others added 3 commits July 24, 2026 10:03

chore: trim comment narration in the drop-in seeding code

6e8fd08

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

PER-8985 feat: Playwright drop-in baseline seeding — build attributes + exec orchestration - #2335

PER-8985 feat: Playwright drop-in baseline seeding — build attributes + exec orchestration#2335
Shivanshu-07 wants to merge 20 commits into
masterfrom
feat/playwright-dropin-baseline

Shivanshu-07 commented Jul 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Shivanshu-07 commented Jul 17, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Shivanshu-07 commented Jul 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Shivanshu-07 commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Depends on

Testing

Deploy ordering

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Shivanshu-07 commented Jul 17, 2026

Claude Code PR Review

Summary

Review Table

Findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Shivanshu-07 commented Jul 27, 2026

Claude Code PR Review

Summary

Review Table

Findings

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Shivanshu-07 commented Jul 13, 2026 •

edited

Loading